Resource-limited Index Construction for Large Texts

An inverted index stores, for each term that appears in a collection of documents, a list of document numbers containing that term. Such an index is indispensable when Boolean or informal ranked queries are to be answered. Construction of the index is, however, a non-trivial task. Simple methods using in-memory data structures cannot be used for large collections because they require too much random access storage, and traditional disk-based methods require large amounts of temporary le space. In this paper a new method is described, making use of two simple compression codes for the positive integers, and an in-place merging structure. Analysis shows that the new technique is capable of inverting a 5 Gb collection in approximately 13 hours on a typical workstation, using 40 Mb of main memory and 120 Mb of additional temporary disk space.